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(57) A motion picture retrieval information storage 
apparatus (40) storing retrieval information for retrieving 
a motion picture, expressed by motion picture data, and 
constituted of one or more partial motion pictures (140. 
142, 144, 146, 160) on the time axis includes a retrieval 
information generating section (102, 103, 104, 105) 
generating retrieval information corresponding to each 
of the one or more partial motion pictures on the basis 
of the motion picture data and a storage section (106) 



storing said retrieval information into a storage medium 
together with a correspondence information between 
the retrieval information and the motion picture data, 
wherein retrieval information generating section (102, 
103, 104, 105) includes a first summary information gen- 
erating section (103) for generating summary informa- 
tion (1 88) of one ormore spatially partial motion pictures 
obtained by spatially splitting each of the one or more 
partial motion pictures (1 40, 1 42, 1 44, 1 46, 1 60) on the 
time axis. 
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Description 

Technical Field 

[0001] The present invention relates to an apparatus 
storing information for retrieving a motion picture (a mo- 
tion picture retrieval information storage apparatus) and 
an apparatus for retrieving a motion picture (a motion 
picture retrieving apparatus), and more particularly, to a 
motion picture retrieval information storage apparatus 
and a motion picture retrieving apparatus, capable of ef- 
ficiently retrieving and taking out a desired scene among 
plural scenes constituting a motion picture. 

Background Art 

[0002] With improvement on performance of a com- 
puter, realization of a storage apparatus having a large 
capacity, expansion of a communication infrastructure 
and other progresses, a service or the like has been 
coming into wide spread use that maintains a data base 
of a large amount of pictures and delivers a necessary 
picture when a necessity arises. Since a technology for 
digitizing a picture has progressed, an individual can 
build a data base of motion pictures. 
[0003] When a data base of motion pictures has been 
built, then how to retrieve a desired picture therefrom 
becomes a problem. The present invention relates to an 
improvement on such a technology for retrieval of a mo- 
tion picture. 

[0004] The term "motion picture" used in the present 
specification includes both types of motion picture data 
not processed and constituted of RGB (Red, Green and 
Blue) signals and motion picture data digitized and cod- 
ed according to a motion picture compression format as 
is in MPEG (Moving Picture Experts Group) or the like. 
[0005] The term "scene" means a constituent unit of 
a motion picture including a consecutive plural picture 
frames and for example, of a motion picture obtained in 
one time shooting from the start to the end of recording 
or a motion picture for which editing has been finished, 
a length of the motion picture partitioned with two edit 
points at both ends thereof. An edit point includes, in a 
case, special effects such as fading and a dissolve. Fur- 
thermore, one scene may include plural scenes. 
[0006] The term "a scene change point" means a po- 
sition on the time axis at which two scenes are switched 
over therebetween and a leading frame directly after a 
scene is switched is referred to as "a scene change 
frame." 

[0007] The term "a key frame" means a distinctive 
frame utilized in retrieval or the like, representing a 
scene or all of a motion picture. 

[0008] A motion picture data is "contents" constituted 
of image information changing along the time axis.. Mo- 
tion picture data becomes an object of retrieval in the 
entirety. Therefore, the simplest method for retrieval of 
a desired motion picture is to select a desired motion 



picture by seeing all of a motion picture. However, it 
takes a time to see all of a motion picture. Therefore, 
when the number of motion pictures registered at a data 
base increases, it takes an extremely long time to take 
5 out a desired motion picture. Accordingly, this method 
is unrealistic. 

[0009] In a generally adopted method for efficient re- 
trieval of a motion picture, various kinds of information 
for retrieval (retrieval information) representing contents 
10 of respective motion pictures are prepared so as to be 
ready for use in retrieval of the respective motion pic- 
tures and retrieval is performed on the basis of the re- 
trieval information. 

[0010] Information used as retrieval information in- 
15 eludes information attached to a picture (attached infor- 
mation), information expressing characteristics of a pic- 
ture (characteristic information) and information ex- 
pressing an outline of a picture (outline information). The 
attached information includes a title or a descriptive text 

20 prepared accompanying a motion picture. The charac- 
teristic information includes information expressing 
characteristics of a picture such as a frequency of a lu- 
minance value derived from a picture signal, motion in- 
formation between frames, or the like. The outline infor- 
ms mation includes information of scene change points, key 
frames, or the like, which are determined by character- 
istic information of a picture or an input from the outside. 
[0011] Of the above-described information, the char- 
acteristic information is especially convenient and im- 

30 portant. For example, a key frame representing a motion 
picture can be selected by using characteristic informa- 
tion. The characteristic information is easily digitized 
and if a characteristic of a desired motion picture is dig- 
itized in the form of characteristic information, it can be 

35 easy to extract a motion picture having a characteristic 
coinciding with a requested characteristic information. 
In the following description, the term "retrieval informa- 
tion" means this characteristic information. 
[0012] In general, a motion picture includes plural 

40 scenes. When a retriever retrieves a motion picture, the 
retrieval is performed in order to find out a desired frame 
or a desired scene included in a motion picture or one 
of a plurality of motion pictures which are an object for 
retrieval. A fundamental method for finding out a desired 

45 frame or a scene in a motion picture using retrieval in- 
formation is to follow a procedure in which retrieval in- 
formation relating to frames or retrieval information re- 
lating to scenes is obtained in advance and a frame or 
a scene is selected on the basis of such retrieval infor- 

50 mation. 

[0013] An apparatus and process are disclosed in 
Japanese Patent Laying-Open No. 9-284702 in connec- 
tion with this point in which a total sum of motion vectors 
of each frame and a continuity of a plurality of small re- 
55 gions constituting a frame, in a frame and between 
frames are adopted as feature of a frame for detection 
of a scene change point. 

[0014] Furthermore, a control method is disclosed in 
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Japanese Patent Laying-Open No. 7-38842 in a case 
where the average of motion vectors in magnitude and 
color histograms of pictures in each scene are used as 
a feature of the scene and a representative picture of 
each scene is displayed as a picture index. 
[0015] If an apparatus for storing information for re- 
trieving a motion picture (a motion picture retrieval in- 
formation storage apparatus) and an apparatus for re- 
trieving a motion picture (a motion picture retrieving ap- 
paratus) are fabricated, taking into consideration such 
prior art techniques, the apparatuses will be as shown 
below. 

[0016] Referring to Fig. 1, a prior art motion picture 
retrieval information storage apparatus which is as- 
sumed includes: an analysis section 601 for analyzing 
inputted motion picture data to divide the data into 
scenes and output information expressing a structure of 
a motion picture such as scene dividing positions (scene 
change); a retrieval information generating section 602 
for a frame unit, connected so as to receive an output 
of analysis section 601 and inputted motion picture data 
and for generating and outputting summary information 
of a picture such as a total sum of motion vectors cor- 
responding to each of frames in a scene as retrieval in- 
formation on the basis of motion picture structure infor- 
mation and motion picture data outputted from analysis 
section 601 ; a retrieval information generating section 
603 for a scene unit, connected so as to receive an out- 
put of retrieval information generating section 602 for a 
frame unit and generating and outputting summary in- 
formation such as the average value of motion vectors 
in magnitude for all of a scene as retrieval information 
on the basis of summary information of a frame unit out- 
putted from retrieval information generating section 602 
for a frame unit; a retrieval information generating sec- 
tion 604 for a motion picture for arranging : into a pre- 
scribed format, motion picture structure information out- 
putted from analysis section 601, retrieval information 
of each frame outputted from retrieval information gen- 
erating section 602 for a frame unit and retrieval infor- 
mation for all of a scene outputted from retrieval infor- 
mation generating section 603 for a scene unit to gen- 
erate and output retrieval information corresponding to 
a motion picture; and a storage section 605 for storing 
original motion picture and the retrieval information cor- 
responding to a motion picture outputted from retrieval 
information generating section 604 for a motion picture, 
in a state of both being related to each other into a stor- 
age medium 606. 

[0017] Note that one frame can be considered to be 
a unit of a motion picture obtained by temporal division 
of the motion picture. Hence, a frame can be called as 
one example of a temporally divided block. 
[001 8] In this specification, a notation of "retrieval in- 
formation of a frame unit" is used for expressing retrieval 
information provided with a frame as a unit. Similarly, a 
notation of "retrieval information of a scene unit" is used 
for expressing retrieval information provided with a 
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scene as a unit. 

[0019] Referring to Fig. 2, a prior art motion picture 
retrieving apparatus which is assumed includes: a read- 
out section 701 for reading out motion picture retrieval 
5 information and motion picture data from storage medi- 
um 606; a retrieval information managing section 702 
for holding each motion picture retrieval information 
supplied from read-out section 701 and receiving a re- 
quest for retrieval information to output retrieval infor- 

10 mation of a scene unit and retrieval information of a 
frame unit together with structure information of a mo- 
tion picture; a retrieval executing section 703 for a scene 
unit for receiving a request for retrieval with a scene as 
a unit to request retrieval information of a scene unit and 

15 motion picture structure information from retrieval infor- 
mation managing section 702 and detecting a scene 
meeting the retrieval request on the basis of the retrieval 
information of a scene unit and the motion picture struc- 
ture information of a scene received to output informa- 

20 tion on the detected scene; a retrieval executing section 
704 for a frame unit for receiving a request for retrieval 
with a frame as a unit to request retrieval information of 
a frame unit in a scene and motion picture structure in- 
formation from retrieval information managing section 

25 702 and detecting a scene or a frame meeting the re- 
trieval request on the basis of the retrieval information 
of a frame unit in a scene and the motion picture struc- 
ture information received to output information on the 
detected scene or frame; a data managing section 705 

30 for taking out and outputting picture data corresponding 
to a detected scene or frame through read-out section 
701 on the basis of information on the detected scene 
or information on the detected frame, outputted from re- 
trieval executing section 703 for a scene unit and retriev- 
es a l executing section 704 for a frame unit: and a retrieval 
control section 706 for responding to a retrieval instruc- 
tion given by an operator to provide a retrieval request 
to retrieval executing section 703 for a scene unit or re- 
trieval executing section 704 for a frame unit, and for 

40 repeating a process displaying motion picture data pro- 
vided from data managing section 705 as a retrieval re- 
suit to output the retrieval result at the end of retrieval. 
[0020] The apparatus shown in Fig. 1 is considered 
to operate as follows: 

45 [0021 ] When motion picture data is inputted, analysis 
section 601 divides motion picture data into scene units. 
Analysis section 601 further outputs motion picture 
structure information. 

[0022] Retrieval information generating section 602 
50 for a frame unit generates summary information corre- 
sponding to a frame in a scene on the basis of motion 
picture structure information outputted from analysis 
section 601 and motion picture data. Retrieval informa- 
tion generating section 602 for frame unit further outputs 
55 the summary information thus generated to retrieval in- 
formation generating section 603 for a scene unit and 
retrieval information generating section 604 for a motion 
picture as retrieval information. 
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[0023] Retrieval information generating section 603 
for a scene unit generates summary information for all 
of a scene on the basis of the summary information of 
a frame unit provided from retrieval information gener- 
ating section 602 for a frame unit. Retrieval information 
generating section 603 for a scene unit further provides 
the summary information thus generated to retrieval in- 
formation generating section 604 for a motion picture as 
retrieval information. 

[0024] Retrieval information generating section 604 
for a motion picture receives motion picture structure in- 
formation, retrieval information of a frame unit and re- 
trieval information for all of a scene, from analysis sec- 
tion 601 , retrieval information generating section 602 for 
a frame unit and retrieval information generating section 
603 for a scene unit : respectively, and generates retriev- 
al information corresponding to a motion picture : arrang- 
ing the information into a prescribed format. Retrieval 
information generating section 604 for a motion picture 
provides the retrieval information corresponding to a 
motion picture thus generated to storage section 605. 
[0025] Storage section 605 stores original motion pic- 
ture data and the retrieval information provided from re- 
trieval information generating section 604 for a motion 
picture into storage medium 606. At this time, storage 
section 605 attaches information indicating that the mo- 
tion picture data and the retrieval information, to be both 
stored there, correspond with each other, to one or both 
of the motion picture data and the retrieval information; 
or stores the information of the correspondence sepa- 
rately from the motion picture data and the retrieval in- 
formation. 

[0026] Referring to Fig. 2, it is assumed that the prior 
art retrieving apparatus operates in the following way: 
At the start of retrieval, an operator gives a retrieval in- 
struction to retrieval control section 706. Retrieval con- 
trol section 706 selects retrieval of a scene unit or re- 
trieval of a frame unit in response to the retrieval instruc- 
tion. 

[0027] It is assumed that, for example, retrieval of a 
scene is selected. Then, retrieval control section 706 
provides a retrieval request to retrieval executing sec- 
tion 703 fora scene unit. Retrieval executing section 703 
for a scene unit responds to this request to request re- 
trieval information of a scene unit and motion picture 
structure information from retrieval information manag- 
ing section 702 and receives the retrieval information of 
a scene unit and motion .picture structure information 
outputted from retrieval information managing section 
702. 

[0028] Retrieval executing section 703 for a scene 
unit detects a scene meeting the retrieval request pro- 
vided from retrieval control section 706 on the basis of 
the retrieval information and the motion picture structure 
information of a scene unit. Retrieval executing section 
703 for a scene unit outputs information on the detected 
scene to data managing section 705. 
[0029] Data managing section 705 reads out corre- 



sponding picture data from storage medium 606 through 
read-out section 701 on the basis of information on a 
scene provided from retrieval executing section 703 for 
a scene unit to provide the picture data to retrieval con- 
5 trol section 706. 

[0030] Retrieval control section 706 displays the pic- 
ture data. The operator sees the displayed picture data 
to determine a next retrieval policy and gives a next re- 
trieval instruction to retrieval control section 706. The 
10 similar process is repeated hereinafter. 

[0031] A picture that the operator desires to take out 
from the data base is taken out in such a way : and then 
the retrieval finishes. A retrieval result is outputted from 
retrieval control section 706. 
15 [0032] In the prior art, retrieval information of a scene 
is generated on the basis of retrieval information of 
frame units included in the scene. This is because of 
general understanding that the minimum unit constitut- 
ing a scene is a frame. 

20 [0033] Retrieval information of a scene is generated 
on the basis of retrieval information of frame units in 
such a way and then spatial features in each frame ob- 
tained by splitting the frame into small regions (retrieval 
information of a screen-split block units) can be used as 

25 retrieval information. However, a problem has been en- 
countered since in a prior art apparatus, spatial features 
of a picture over all of a scene having a length on the 
time axis cannot be handled. In the present specifica- 
tion, such spatial features of a picture over all of a scene 

30 is referred to as "retrieval information of a spatially split 
block unit." 

[0034] Consider for example, a case where a scene 
is taken out that has a feature that an object in a vigorous 
motion is displayed in a middle portion of a screen. In 

35 the prior art, it was not able to take out such a scene 
from retrieval information of a scene unit but a necessity 
arose for retrieval using retrieval information attached 
to each of frames included in a scene. For this reason, 
in the prior art such retrieval was performed with a very 

40 low efficiency. 

[0035] The present invention has been made in light 
of the above-described problem and it is an object of the 
present invention to provide a motion picture retrieval 
information storage apparatus and a motion picture re- 

^5 trieving apparatus capable of efficiently performing re- 
trieval of a motion picture using a new index expressing 
a feature of a scene with correctness. 
[0036] It is another object of the present invention to 
provide a motion picture retrieval information storage 

50 apparatus and a motion picture retrieving apparatus ca- 
pable of efficiently performing retrieval of a motion pic- 
ture by extracting not only a feature of a scene by divid- 
ing a scene into blocks on the time axis but also another 
feature by splitting the scene only spatially without di- 

55 viding the scene on the time axis. 

[0037] It is still another object of the present invention 
to provide a motion picture retrieval information storage 
apparatus and a motion picture retrieving apparatus ca- 
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pable of efficiently performing retrieval of a motion pic- 
ture using a new index expressing a feature of a scene 
with correctness, with an small amount of information 
for the retrieval. 

5 

Disclosure of the Invention 

[0038] A motion picture retrieval information storage 
apparatus relating to the present invention includes: a 
retrieval information generating section generating re- '0 
trieval information corresponding to each of one or more 
partial motion pictures on the time axis : on the basis of 
motion picture data and motion picture structure infor- 
mation that divides a motion picture expressed by the 
motion picture data on the time axis into one or more. '5 
partial motion pictures on the time axis; and a storage 
section storing the retrieval information and the motion 
picture structure information into a storage medium to- 
gether with the corresponding motion picture data. The 
retrieval information generating section includes a first 20 
summary information generating section for generating 
summary information of one ormore spatially partial mo- 
tion pictures obtained by spatially splitting each of one 
or more partial motion pictures on the time axis. 
[0039] Summary information is generated from spa- 25 
tially partial motion pictures obtained by spatially split- 
ting each of partial motion pictures on the time axis. This 
summary information has been introduced into the 
present invention in the art for the first time and is not 
information obtained from a temporal part of the partial 30 
motion pictures on the time axis. This summary infor- 
mation is obtained from a spatially partial motion picture 
being only a part of the motion picture on the spatial do- 
main, but residing over all of the partial motion picture 
on the time axis. Therefore, this summary information 35 
expresses a spatial feature of the partial motion pictures 
over all of the partial motion picture on the time axis. By 
use of the new index, a retrieval is efficiently performed 
with more of emphasis placed on a spatial feature of the 
partial motion picture than on a temporal feature thereof. 40 
[0040] A motion picture retrieving apparatus relating 
to another aspect of the present invention is a motion 
picture retrieving apparatus for retrieving a desired pic- 
ture using retrieval information corresponding to each 
of one or more partial motion pictures constituting a mo- 
tion picture. Motion picture data expressing the motion 
picture is related with retrieval information and the re- 
trieval information includes summary information of one 
or more spatially partial motion pictures obtained by 
spatially splitting each of one or more partial motion pic- so 
tures on the time axis. A motion picture retrieving appa- 
ratus relating to the present invention includes: a infor- 
mation extracting section for reading and managing re- 
trieval information; and a first partial motion picture re- 
trieving section, connected with the information manag- 55 
ing section, and for retrieving a partial motion picture on 
the time axis meeting a first retrieval request in response 
to the first retrieval request with a partial motion picture 
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on the time axis as a retrieval unit, provided from the 
outside, using summary information of a spatially partial 
motion picture included in the retrieval information. 
[0041 ] Summary information obtained from a spatially 
partial motion picture obtained by spatially splitting each 
of partial motion pictures on the time axis has been new- 
ly introduced into the present invention in the art and is 
obtained from a spatially partial motion picture being on- 
ly a part of the motion picture on the spatial domain : but 
residing over all of the partial motion picture on the time 
axis. Hence, this summary information expresses a spa- 
tial feature of the partial motion pictures over all of the 
partial motion picture on the time axis. Therefore : ac- 
cording to a motion picture retrieving apparatus relating 
to the present invention, by use of the new index, a re- 
trieval can be efficiently performed with more of empha- 
sis placed on a spatial feature of the partial motion pic- 
ture than on a temporal feature thereof. 

Brief Description of the Drawings 

[0042] 

Fig. 1 is a schematic block diagram of an imaginary 
prior art motion picture retrieval information storage 
apparatus; 

Fig. 2 is a schematic block diagram of an imaginary 
prior art motion picture retrieving apparatus; 
Fig. 3 is a block diagram of a motion picture retrieval 
information storage apparatus relating to one em- 
bodiment of the present invention; 
Fig. 4 is a representation showing a process in 
which a motion picture is divided into scenes in the 
motion picture retrieval information storage appara- 
tus relating to the one embodiment of the present 
invention: 

Fig. 5 is a representation showing a process in 
which retrieval information of a temporally divided 
block unit is generated in the motion picture retrieval 
information storage apparatus relating to the one 
embodiment of the present invention; 
Fig. 6 is a representation showing a process in 
which retrieval information of a spatially split block 
unit is generated in the motion picture retrieval in- 
formation storage apparatus relating to the one em- 
bodiment of the present invention; 
Fig. 7 is a representation for describing a concept 
of retrieval information of a scene unit in the motion 
picture retrieval information storage apparatus re- 
lating to the one embodiment of the present inven- 
tion; 

Fig. 8 is a representation for describing a concept 
of retrieval information of a spatially split block unit 
in the motion picture retrieval information storage 
apparatus relating to the one embodiment of the 
present invention; 

Fig. 9 is a representation for describing a concept 
of retrieval information in a spatially split block unit 
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excluding redundancy, in the motion picture retriev- 
al information storage apparatus relating to the one 
embodiment of the present invention; 
Fig. 10 is a block diagram of a motion picture re- 
trieving apparatus relating to the one embodiment 
of the present invention; 

Fig. 11 is a pictorial diagram showing a scene of 
reference in similarity retrieval using scene unit re- 
trieval information; 

Fig. 12 is a pictorial diagram showing one scene in 
similarity retrieval using scene unit retrieval infor- 
mation; 

Fig. 1 3 is a pictorial diagram showing another scene 
in similarity retrieval using scene unit retrieval infor- 
mation; 

Fig. 14 is a pictorial diagram showing a scene of 
reference in similarity retrieval using spatially split 
block unit retrieval information: 
Fig. 15 is a pictorial diagram showing one scene in 
similarity retrieval using spatially split block unit re- 
trieval' info rmation; 

Fig. 1 6 is a pictorial diagram showing another scene 
in similarity retrieval using spatially split block unit 
retrieval information; 

Fig. 1 7 is a representation for showing a concept of 
storage of scene unit retrieval information and se- 
lective storage of retrieval information of a spatially 
split block unit in the motion picture retrieval infor- 
mation storage apparatus relating to the one em- 
bodiment of the present invention; 
Fig. 18 is a table showing a meaning of a value of 
a splitting number specifying flag; 
Fig. 19 is a representation expressing a concept of 
storage of scene unit retrieval information and hier- 
archical storage of retrieval information of a spatial- 
ly split block unit in the motion picture retrieval in- 
formation storage apparatus relating to the one em- 
bodiment of the present invention; 
Fig. 20 is a representation showing an example 
splitting when a frame is split into 2 2 parts; 
Fig. 21 is a representation showing an example 
splitting when a frame is split into 2 3 parts; 
Fig. 22 is a representation showing an example 
splitting when a frame is split into 2 4 parts: 
Fig. 23 is a representation showing an example 
splitting when a frame is split into 2 5 parts; 
Fig. 24 is a representation showing an example 
splitting when a frame is split into 2 6 parts: and 
Fig. 25 is a representation showing a motion picture 
data base in which motion picture data is stored be- 
ing divided into scene units. 

Best Mode for Carrying out the Invention 

[0043] Fig. 3 shows a block diagram of a motion pic- 
ture retrieval information storage apparatus 40 relating 
to the embodiment. Referring to Fig. 3, motion picture 
retrieval information storage apparatus 40 includes: an 



analysis section 101 for receiving motion picture data 
and analyzing a structure of a motion picture to divide 
the data into scenes and output motion picture structure 
information such as scene change points: a first retrieval 
5 information generating section 1 02 for receiving the mo- 
tion picture data and the motion picture structure infor- 
mation outputted from analysis section 101 , and gener- 
ating and outputting summary information of picture da- 
ta of each of frames included in a scene (summary in- 
fo formation with a temporally divided block called as a 
frame as a unit), a second retrieval information gener- 
ating section 103 for receiving the motion picture struc- 
ture information outputted from analysis section 1 01 and 
the motion picture data, and generating and outputting 
15 summary information of a spatially split block constitut- 
ed of picture data at the same split block position of all 
of the frames in a scene as retrieval information of each 
of the scenes; a third retrieval information generating 
section 104 for receiving the summary information of 
20 each of the frames (temporally divided blocks) in a 
scene outputted from first retrieval information generat- 
ing section 1 02 and/or the summary information of each 
of the spatially split blocks in a scene outputted from 
second retrieval information generating section 103 as 
25 inputs, generating and outputting summary information 
of all of the scene on the basis of the summary informa- 
tion from first retrieval information generating section 
102 and/or the summary information from second re- 
trieval information generating section 103; a fourth re- 
30 trieval information generating section 105 for arranging 
the summary information outputted from first retrieval in- 
formation generating section 102, the summary infor- 
mation outputred from second retrieval information gen- 
erating section 103, the summary information outputted 
35 from third retrieval information generating section 104 
and the motion picture structure information outputted 
from analysis section 101 into a prescribed format to 
thereby generate and output retrieval information corre- 
sponding to the motion picture; and a storage section 
40 106 for receiving the retrieval information of the motion 
picture outputted from fourth retrieval information gen- 
erating section 105 and the original motion picture data 
and storing correspondence information between the 
retrieval information and the original motion picture data 
45 orthe like attaching the corresponding information orthe 
like to one or both of the retrieval information and the 
original motion picture data : or the correspondence in- 
formation orthe like as separate data from the retrieval 
information and the original motion picture, into a stor- 
50 age medium 107. 

[0044] Motion picture data that an apparatus of the 
present invention handles may be any of original picture 
signals such as RGB signals, and picture data encoded 
according to a motion picture compression format such 
55 as MPEG (Moving Picture Experts Group) or the like. In 
the embodiment, it is assumed for the sake of conven- 
ience in description that the motion picture data is en- 
coded data. 
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[0045] More detailed description will be given of func- 
tions of blocks shown in Fig. 3 below: Analysis section 
101 divides a motion picture into temporally small units 
(scenes). For this purpose, analysis section 101 per- 
forms detection of scene change points in motion picture 
data. Analysis section 101, in this example, detects a 
scene change point using frequency information in a 
frame in predictive mode of coded blocks included in en- 
coded picture data. 

[0046] For example, referring to Fig. 4, analysis sec- 
tion 1 01 detects a first scene change frame 1 30 : a sec- 
ond scene change frame 132, a third scene change 
frame 134 and a fourth scene change frame 136 of a 
motion picture 120. Thereby, analysis section 101 di- 
vides motion picture 1 20 into a first scene 1 40 constitut- 
ed of picture data between first scene change frame 1 30 
and second scene change frame 132 (including first 
scene change frame 130), a second scene 142 consti- 
tuted of picture data between second scene change 
frame 1 32 and third scene change frame 1 34 (including 
second scene change frame 132), a third scene 144 
constituted of picture data between third scene change 
frame 134 and fourth scene change frame 136 (includ- 
ing third scene change frame 134), and a fourth scene 
146 constituted of picture data from fourth scene change 
frame 136 to the last frame of motion picture 120 (in- 
cluding fourth scene change frame 136). That is, anal- 
ysis section 101 divides motion picture 120 into four 
scenes 140, 142 : 144 and 146, in this example. 
[0047] Analysis section 101 has a function of output- 
ting information expressing positions in the picture data 
on first scene change frame 1 30, second scene change 
frame 1 32, third scene change frame 1 34, fourth scene 
change frame 1 36 and so on, as motion picture structure 
information. 

[0048] Referring to Fig. 5, first retrieval information 
generating section 102 performs the process as de- 
scribed below on a scene 160 obtained by dividing mo- 
tion picture data on the basis of motion picture structure 
information outputted from analysis section 101 to pre- 
pare summary information of each of frames included 
in scene 160. Note that scene 160 includes picture data 
from a frame 162A to a frame 162N. 
[0049] First retrieval information generating section 
1 02 obtains frequency information 1 66A to 1 66N of mo- 
tion vectors in respective frames 1 62A to 1 62N. Further- 
more, first retrieval information generating section 102 
obtains averages 1 68A to 1 68N of motion vectors in re- 
spective frames 162A to 162N. 

[0050] Frequency information 166 and average infor- 
mation 168 thus obtained constitute a summary infor- 
mation 1 70 of each frame here. The summary informa- 
tion is given as a function of a frame number (that is a 
function of time). 

[0051] The summary information is retrieval informa- 
tion obtained by compression of information spatially 
distributed in each of the frames included in scene 1 60. 
A frame is a kind of a temporally divided block since a 



frame is a unit obtained by temporally dividing a motion 
picture. Hence, summary information 170 can be said 
retrieval information of a temporally divided block unit. 
[0052] Second retrieval information generating sec- 
5 tion 103 receives the motion picture structure informa- 
tion (information on scene change positions) outputted 
from analysis section 101 and the motion picture data. 
Second retrieval information generating section 103 
generates summary information of a spatially split block 
'0 using the motion picture structure information and the 
motion picture data in the following way. 
[0053] Note that the spatially split block here is a block 
as described below: All of frames in one scene are each 
spatially split into plural split blocks in the same way. All 
15 of plural split blocks at corresponding positions on all of 
the respective frames are collectively called a spatially 
split block of the scene. Therefore, when one scene in- 
cludes n frames and each frame is spatially split into m 
split blocks, the scene is split into m spatially split blocks 

20 and each spatially split block includes n split blocks. 
[0054] Specifically, referring to Fig. 6 : in second re- 
trieval information generating section 1 03, the following 
process is applied to scene 1 60. As described previous- 
ly, it is assumed that scene 160 includes a plurality of 

25 frames 162A to 162N. Frames 162A to 162N are each 
spatially split into M split blocks, wherein a split block 
means each of small regions obtained when a picture 
plane is split into a plurality of parts. For example, frame 
1 62A shown in Fig. 6 is split into a plurality of split blocks 

30 180A1 to 180AM. This applies to each of the other 
frames 162B to 162N. 

[0055] One spatially split block of scene 1 60 is formed 
from split blocks at the same position on the plurality of 
frames 162A to 162N. For example, spatial split block 
35 182-M is formed by a collection of the Mth split blocks 
of respective N frames (the number of elements is equal 
to N). The other spatial split blocks are formed in a sim- 
ilar way. 

[0056] Summary information 188 is generated by 
40 compressing information distributed in the direction of 
the time axis for each of the M spatially split blocks thus 
obtained. Summary information 1 88 referred to herein 
is summary information of each spatially split block and 
a value of a function of a position (coordinates) in a 
45 frame of a split block included in each of the spatially 
split blocks. 

[0057] Typically, summary information 188 includes 
frequency information 184 and average information 
1 86. The frequency information in a spatial split block in 

50 a predictive mode of each encoded block or frequency 
information in a spatially split block of motion vectors in 
motion compensative prediction encoding can be used 
as frequency information 184. For example, frequency 
information 1 84A is obtained from a spatially split block 

55 182-1, though not shown, and frequency information 
184B is obtained from a spatially split block 182-2 : 
though not shown and the other frequency information 
is obtained in a similar way; finally, frequency informa- 
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tion 1 84M is obtained from a spatially split block 1 82-M. 
[0058] A total sum or the average of motion vectors 
in a spatially split block can be used as average infor- 
mation 186. For example, average information 186A is 
obtained from a spatially split block 182-1 , not shown : 
average information 186B is obtained from a spatially 
split block 182-2, not shown, and the other average in- 
formation is obtained in a similar way; finally, average 
information 1 86M is obtained from a spatially split block 
182-M. 

[0059] In addition to the total sum and the average, 
as summary information, there also can be used a 
standard deviation or the like of motion vectors in a spa- 
tially split block. 

[0060] Third retrieval information generating section 

104 receives summary information outputted from first 
retrieval information generating section 102 and sum- 
mary information outputted from second retrieval infor- 
mation generating section 103, and generates and out- 
puts summary information of all of the scene on the ba- 
sis of one or both of the summary information. 
[0061] The summary information of all of a scene 
herein is retrieval information used in retrieval of the 
scene. As summary information of all of a scene, there 
can be used frequency information, the average, a 
standard deviation or the like of all of the scene with re- 
spect to motion vectors, luminance values, color differ- 
ences or the like parameter. 

[0062] In this specification, a notation of "retrieval in- 
formation of a temporally divided block unit (frame unit)" 
is used for expressing retrieval information provided 
with a temporally divided block (frame) as a unit. Simi- 
larly, a notation of "retrieval information of a spatially 
split block unit" and "retrieval information of a scene 
unit" are used for expressing retrievaf information pro- 
vided with a spatially split block and a scene as a unit. 
[0063] Fourth retrieval information generating section 

105 arranges summary information outputted from first 
retrieval information generating section 102, summary 
information outputted from second retrieval information 
generating section 103, summary information outputted 
from third retrieval information generating section 1 04 
and motion picture structure information outputted from 
analysis section 101 into a prescribed format to gener- 
ate and output retrieval information corresponding to the 
motion picture and provide the retrieval information to 
storage section 106. 

[0064] Storage section 106 receives the retrieval in- 
formation corresponding to the motion picture, output- 
ted from fourth retrieval information generating section 
1 05 and the original motion picture data to store only the 
retrieval information or the retrieval information and the 
motion picture data into storage medium 107. At this 
time, storage section 106 stores information showing 
that the motion picture data and the retrieval information 
corresponds with each other, attaching to one or both 
of, or as separate data from, the motion picture data and 
the retrieval information. 



[0065] Information of a magnitude of moiion obtained 
on the basis of motion vectors included in motion com- 
pensative prediction-encoded motion picture data is 
used, as retrieval information adopted in a motion pic- 

5 ture retrieval information storage apparatus 40 relating 
to the embodiment described above in connection to a 
structure and a general operation thereof. 
[0066] Specifically, retrieval information of a tempo- 
rally divided block (frame) unit is obtained for each of 

10 frames in a scene, as a total sum of the absolute values 
of motion vectors in all of a frame, obtained from the 
motion vectors attached to each of encoded blocks. 
Note that the retrieval information may be obtained as 
the average of the absolute values of motion vectors in- 

'5 stead of the total sum of the absolute values of motion 
vectors with the same effect. Actually, if the number of 
encoded blocks included in one frame is determined, ei- 
ther a total sum or the average of the absolute values 
of motion vectors can be obtained from the other with 

20 ease. 

[0067] A total sum of the absolute values of motion 
vectors in each of spatially split blocks is used as re- 
trieval information of a spatially split block unit. Herein, 
too, the average of the absolute values of motion vectors 

25 can be used instead of the total sum of motion vectors. 
In a case where split blocks constituting a spatially split 
block coincides with encoded blocks, the number of mo- 
tion vectors corresponding to each of the split blocks is 
1 . In a case where each split block includes a plurality 

30 of encoded blocks, the number of motion vectors corre- 
sponding to each of the split blocks is equal to the 
number of encoded blocks included in each split block. 
[0068] Alternatively, retrieval information of each spa- 
tially split block unit may be a total sum of the absolute 

35 values of motion vectors in each spatially split block di- 
vided by the number of split blocks (on a frame) consti- 
tuting each spatially split block. The number of the split 
blocks in this case is equal to the number of frames con- 
stituting a scene. Furthermore, the retrieval information 

<to of each spatially split block unit may be a total sum of 
the absolute values of motion vectors in each spatially 
split block unit divided by the number of coded blocks 
included in split blocks (on a frame) constituting each 
spatially split block. 

45 [0069] If a total sum of the absolute values of motion 
vectors is adopted as retrieval information of a tempo- 
rally divided block or a spatially split block, a total sum 
of the absolute values of all of motion vectors included 
in a scene can be calculated. In the embodiment, the 

so total sum thus obtained is used as retrieval information 
of a scene unit. 

[0070] Then, description will be given of a specific ex- 
ample of a split block for use in generating retrieval in- 
formation of a spatial split block unit. As shown in Fig. 
55 7 t jt is assumed that scene 1 60 is constructed of frames 
1 62A to 1 62N. Scene unit retrieval information I of scene 
160 is obtained from a total sum of the absolute values 
of motion vectors of frames 162A to 162N. 
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[0071] As shown in Fig. 8, it is assumed that frames 
162A to 162N are each split into 2x2 split blocks. That 
is, frames 1 62A to 1 62N are each split into 4 split blocks. 
For example, frame 162A is split into 4 split blocks 
190A1 to 190A4. This applies to the other frames, not 5 
shown, in a similar manner. For example, frame 162B 
is split into frames 1 90B1 to 190B4. Accordingly, as can 
be seen from the previous description of a spatially split 
block, scene 160 is split into 4 spatially split blocks and 
retrieval information ^ to l 4 of spatially split block units 10 
is obtained from a total sum of the absolute values of 
motion vectors in the respective 4 spatially split blocks : 
all as real numbers. 

[0072] Accordingly, as scene retrieval information for 
scene 160. {I, l 1f ! 2 , l 3 , l 4 } obtained by arranging retrieval '5 
information I, \ A , l 2: l 3 , l 4 in ascending order can be used. 
The retrieval information has only to be stored together 
with the original picture data. 

[0073] However, the above-described retrieval infor- 
mation includes redundancy. In this example, this 20 
means that scene unit retrieval information I is equal to 
the sum of retrieval information of spatial split block 
units. For example, scene unit retrieval information I can 
be obtained from retrieval information l n to l 4 of spatially 
split block units. Furthermore, one of retrieval informa- 25 
tion of spatially split block units can be calculated from 
the other three of retrieval information I., to l 4 of spatially 
split block units and scene unit retrieval information I. 
[0074] Therefore, in an apparatus of the embodiment, 
an operation is performed such that l 4 is not included in 30 
retrieval information. Moreover, in an apparatus of the 
embodiment, a value of each of the spatially split block 
unit retrieval information is shown as a percent thereof 
relative to retrieval' information of a scene unit as 100 
without using a value of an original value of each spa- 35 
tially split block unit retrieval information as it is. If ratios 
of the first to third spatially split block retrieval informa- 
tion to scene unit retrieval information I are assumed as 
P 1 and P 2 and P 3 , retrieval information of spatially split 
block units at this time are obtained in a way as shown 40 
in Fig. 9. That is, retrieval information P 1 of the first spa- 
tially split block is calculated from retrieval information 
of the first spatially split block and the scene unit. Re- 
trieval information P 2 of the second spatially split block 
is calculated from retrieval information of the second 
spatially split block and the scene unit. Retrieval infor- 
mation P 3 of the third spatially split block is calculated 
from retrieval information of the third spatially split block 
and the scene unit. Since redundancy arises if retrieval 
information of the fourth spatially split block is to be ob- 50 
tained, the retrieval information of the fourth spatially 
split block is not obtained. That is, scene 1 60 is split into 
4 spatially split blocks and retrieval information of 3 ones 
of the 4 blocks has only to be obtained. In general, P k • 
is obtained from an equation P k = 100 x \^t\. 55 
[0075] If retrieval information of spatially split blocks 
is stored as percents relative to the scene unit retrieval 
information as 100, advantages as described below can 



be attained. Firstly, since a dynamic range of a signal 
for storing retrieval information is restricted retrieval in- 
formation can be stored with more of efficiency. That is, 
an information amount to be stored decreases com- 
pared with a case of being not stored as a percent. Sec- 
ondly, since a value of retrieval information is normal- 
ized, retrieval information attached to different scenes 
can be compared with each other. Thirdly, as described 
already, one variable to be stored can be omitted. 
[0076] In such a way, motion picture retrieval informa- 
tion is generated by fourth retrieval information gener- 
ating section 105 and stored into storage medium 107 
through storage section 106. 

[0077] Description will be given of an apparatus re- 
trieving a motion picture by use of motion picture data 
and retrieval information stored in storage medium 107 
with the help of motion picture retrieval information stor- 
age apparatus 40 shown in Fig. 3 below: Referring to 
Fig. 1 0, a motion picture retrieving apparatus 50 relating 
to the embodiment includes: a read-out section 501 for 
reading motion picture data and retrieval information 
therefor stored in storage medium 107; a retrieval infor- 
mation managing section 502 for holding the retrieval 
information of a motion picture read out by read-out sec- 
tion 501 to output scene unit retrieval information spa- 
tially split block unit retrieval information or a temporally 
divided block unit retrieval information, in response to a 
retrieval request, together with motion picture structure 
information: a first retrieval executing section 503 for is- 
suing a request to retrieval information managing sec- 
tion 502 in response to a scene unit retrieval request, 
receiving retrieval information of all of the scene and mo- 
tion picture structure information from retrieval informa- 
tion managing section 502 and detecting a scene meet- 
ing the retrieval request for a scene unit to output infor- 
mation of the detected scene; a second retrieval exe- 
cuting section 504 for issuing a request to retrieval in- 
formation managing section 502 in response to a retriev- 
al request for a spatially split block unit and detecting a 
scene meeting the retrieval request on the basis of sum- 
mary information of each of spatially split blocks in a 
scene and motion picture structure information received 
from retrieval information managing section 502 to out- 
put information of the detected scene; a third retrieval 
executing section 505 for issuing a request to retrieval 
information managing section 502 in response to a re- 
trieval request for a temporally divided block unit and 
detecting a scene or a frame meeting the retrieval re- 
quest on the basis of summary information of each of 
frames in a scene and motion picture structure informa- 
tion received from retrieval information managing sec- 
tion 502 to output information of the detected scene or 
frame; a data managing section 506 for holding motion 
picture information read out by read-out section 501 and 
outputting picture data of a scene or a frame detected 
on the basis of a retrieval result outputted from first re- 
trieval executing section 503, second retrieval executing 
section 504 or third retrieval executing section 505; and 
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a retrieval control section 507 for issuing a retrieval re- 
quest for a scene unit to first retrieval executing section 
503, a retrieval request for a spatially split block unit to 
second retrieval executing section 504 or a retrieval re- 
quest for a temporally divided block unit to third retrieval 
executing section 505 in response to inputting of a re- 
trieval instruction from an operator, as a result displaying 
picture data of a scene or a frame outputted from data 
managing section 506 and outputting a retrieval result 
when a retrieval finishes. 

[0078] This motion picture retrieving apparatus 50 op- 
erates as described below: At the start of retrieval, an 
operator gives a retrieval instruction to retrieval control 
section 507. Retrieval control section 507 selects one 
of retrieval in scene units, retrieval in spatially split block 
units or retrieval in temporally divided block units and 
issues a retrieval request to a corresponding one of first 
retrieval executing section 503, second retrieval execut- 
ing section 504 and third retrieval executing section 505. 
[0079] The corresponding one having received the re- 
trieval request from retrieval control section 507 among 
first retrieval executing section 503, second retrieval ex- 
ecuting section 504 and third retrieval executing section 
505 detects a scene or a frame meeting the retrieval re- 
quest on the basis of motion picture structure informa- 
tion, summary information of a scene unit, a spatially 
split block unit or a temporally divided block (frame) unit 
provided from retrieval information managing section 
502 to provide information on the detected scene or 
frame to data managing section 506. 
[0080] Data managing section 506 extracts a picture 
data of a scene or a frame from motion picture data, 
wherein the picture of a scene or a frame corresponds 
to information of detected scene or frame having re- 
ceived from first retrieval executing section 503, second 
retrieval executing section 504 or third retrieval execut- 
ing section 505 to provide the picture data of a scene or 
frame to retrieval control section 507. 
[0081] Retrieval control section 507 displays the pic- 
ture provided from data managing section 506. The op- 
erator determines whether or not the picture is a desired 
scene to issue an instruction such that a retrieval result 
is outputted if the picture is the desired scene to retrieval 
control section 507.. Retrieval control section 507 out-, 
puts the retrieval result in response to this instruction. 
[0082] If the picture displayed by retrieval control sec- 
tion 507 is not one to express the desired scene, the 
operator gives a next retrieval instruction to retrieval 
control section 507. Retrieval control section 507 here- 
inafter performs a similar operation of the above-de- 
scribed to display a scene as a retrieval result. The op- 
eration is repeated till the retrieval finishes. 
[0083] Description will be given of a concrete example 
of motion picture retrieval. In this embodiment, there is 
used information on a magnitude of a motion obtained 
on the basis of motion vectors included in motion picture 
data subjected to motion compensation prediction en- 
coding. 



[0084] A desired scene or a desired frame in a scene 
is retrieved as described below: 

When using retrieval information in scene units, se- 
5 lection can be effected between a scene large in 

motion as a whole and a scene small in motion as 
a whole. 

[0085] When using retrieval information in spatially 
io split block units, selection can be effected , for example, 
between a scene having motions unevenly distributed 
in a space across the scene and a scene having motions 
comparatively evenly distributed in a space across the 
scene. Furthermore, it can be specified what portion of 
>5 a picture motions are localized in. For example, selec- 
tion can be effected on a scene having a portion (a sub- 
ject to be shot) in vigorous motion in the middle part of 
a picture, a scene having a portion in vigorous motion 
in lower half of a picture (a picture including a portion of 
20 sky) or the like. 

[0086] In the embodiment, retrieval information with a 
spatially split block, which was not conceived in the prior 
art as a unit is used in retrieval of a scene. Therefore, 
the following effect is achieved. 
25 [0087] Referring to Figs. 11 to 13, a case is consid- 
ered where a similarity to a scene of reference shown 
in Fig. 11(A) is calculated on a scene A shown in Fig. 
1 2(A) and a scene B shown in Fig. 1 3(A). With such sim- 
ilarity calculation adopted, a scene having a motion fea- 
30 ture analogous to the scene of reference can be detect- 
ed. Figs. 12 to 13 show cases in each of which only re- 
trieval information of a scene unit is used. 
[008~8] In (B) parts of Figs. 11 to 13, there are shown 
scene unit retrieval information for pictures shown in re- 
35 spective (A) parts of the figures. As shown in the (A) 
parts of Figs. 11 to 13, a scene A and a scene B have a 
similar intensity of motion in the entirety thereof. For this 
reason, as shown in the (B) parts of Figs. 11 to 1 3, scene 
unit retrieval information of each of the scenes is of a 
40 similar value. That is, scenes A and B are determined 
to be both analogous to the scene of reference. 
[0089] Subsequent to this, referring to Figs. 14 to 16, 
description will be given of a feature of similarity retrieval 
in a case where retrieval information in spatially split 
45 block units are adopted. In (A) parts of Figs. 14 and 16 : 
there are shown a scene of reference, scenes A and B ; 
respectively. As shown in the figures, each frame of 
each of the scenes has four split blocks. Spatially split 
blocks unit retrieval information P 1: P 2 and P 3 are cal- 
50 culated for spatially split blocks consisted of respective 
three ones among the four split blocks of each scene. 
Furthermore, as shown in (B) parts of Figs. 14 to 16 : 
retrieval information in scene units of a scene of refer- 
ence, scenes A and B is of almost the same value. 
55 [0090] As shown in (C) parts of Figs. 14 to 16 : how- 
ever, the following facts are found when the scenes are 
compared with each other in terms of retrieval informa- 
tion in spatially split block units. That is : scene A has a 
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distribution more similar to a distribution of a spatially 
split block unit retrieval information of the scene of ref- 
erence. This is because in scene B, a person is shifted 
to the right side, so uneven motions are observed spa- 
tially. The scene of reference and scene A both have a 
person almost in the middle of a picture. Therefore : 
while scene A is detected as a scene analogous to the 
scene of reference, scene B is not detected so. That is : 
even in a case where scenes A and B are analogous to 
the scene of reference in motion as a whole, scenes can 
be discriminated therebetween according to whether or 
not motions are unevenly distributed. . 
[0091] Furthermore, if retrieval information in frame 
(temporally divided block) units is utilized, for example, 
selection can be effected between a frame as in vigor- 
ous-motion and a frame as in to the contrary, no motion 
in a scene. 

[0092] By combining retrieval information in scene 
units, retrieval information in spatially split block units 
and retrieval information in temporally divided block 
units, retrieval of a desired scene can be performed in 
more detailed manner. 

[0093] For example, motion picture retrieval can be 
performed in such a way that scenes large in motion as 
a whole are selected fewer scenes having a middle por- 
tion in vigorous motion are further selected and further 
fewer scenes or frames are taken out with a frame in 
especially vigorous motion as a key frame. 
[0094] As described above, according to an appara- 
tus of the embodiment, a desired scene picture can be 
retrieved efficiently on the basis of pictorial features 
such as motions of a picture by specifying conditions 
from multiple aspects. Furthermore, a time required for 
retrieval is also reduced. 

[0095] The above-described embodiment is only one 
example for carrying out the present invention. Various 
modifications or alterations of the apparatuses can be 
performed in addition to the embodiment. 
[0096] For example, in an apparatus of the embodi- 
ment; analysis section 101 automatically finds scene 
change points through signal analysis on motion picture 
.data. The present invention, however, is not limited to 
this, but another procedure may be adopted in which for 
example, an operator sees a motion picture and analyz- 
es a structure of the motion picture based on his or her 
personal point of view to attach a tag to a dividing posi- 
tion for dividing the motion picture into scenes. In this 
case, while analysis section 101 analyzes motion pic- 
ture data in a similar manner, the motion picture is di- 
vided into scenes seeing tags attached to the motion 
picture data. 

[0097] In the above-described embodiment, frequen- 
cy information in frame of motion vectors in motion com- 
pensation prediction encoding is used as frequency in- 
formation 1 66 of summary information of a frame. In the 
present invention, however, no limitation is imposed on 
this, but frequency information in frame in a predictive 
mode of each of coded blocks can also be used as fre- 



quency information. 

[0098] Furthermore, as summary information, there 
may also be used a total sum or a standard deviation of 
motion vectors in frame, or the average luminance or 
5 the average color difference in frame calculated using 
an original picture signal or a decoded picture signal in 
addition to the above-described. 

[0099] A method and procedure for retrieval are not 
limited to combinations shown here. Another possibility 
io is to change the order in which candidates to be selected 
are screened. Moreover a method and procedure for 
retrieval changes according to retrieval information to 
be generated as well. 

[0100] In addition, for example, in an apparatus of the 
is above-described embodiment, there was adopted a col- 
lection across one scene of split blocks each obtained 
by splitting a frame into 2x2 = 4 parts as a spatially split 
block. The number of split parts is not limited to four. For 
example, as shown in Fig. 17, possibility is to split a 

20 frame into 4x4, 8x8 or a number larger than the latter. 
In a case of 4 x 4 splitting, spatially split block unit re- 
trieval information are expressed by a percent ranges 
from P 1 to P 15 . P 16 can be obtained by subtracting the 
sum of retrieval information of the other spatially split 

25 blocks from 100 %. In a case of 8 x 8 splitting, spatially 
split block unit retrieval information are expressed by a 
percent ranges P 1 to P53. P^ can also be obtained by 
subtracting the sum of retrieval information of the other 
spatially split blocks from 1 00 %. 

30 [0101] Moreover, any of a plurality of splitting methods 
such as above-described may be selectively used. In 
this case, spatially split blocks are each constructed of 
split blocks each split according to the number of parts 
selected (, which can be called as "resolution.") to obtain 

35 retrieval information of each spatially split block. Be- 
sides, when retrieval information is stored, in addition to 
scene unit retrieval information and retrieval information 
in spatial split block units according to a selected reso- 
lution, a splitting number specifying flag with corre- 

40 spondence between the value thereof and the splitting 
number shown in Fig. 18 is attached to retrieval infor- 
mation. If the splitting number specifying flag is "0," re- 
trieval information in spatially split block units {P 1r P 2 , 
P 3 } is stored; if the splitting number specifying flag 

45 is "1." retrieval information of spatially split block units 
{P-,, P 2 : P 15 } is stored; if the splitting number speci- 
fying flag is "2," retrieval information of spatially split 
block units {P n> P 2 , — , P 63 } is stored and if the splitting 
number specifying flag is "3," retrieval information of 

50 spatially split block units {P^ P 2 , P 255 } is stored. 
[01 02] By enabling resolutions of spatially split blocks 
to be specified in a plurality of ways, the following effects 
arise: 

55 in a data base in which only a comparatively simple 

and easy retrieval is required in which a rough dis- 
tribution of motions of a motion picture, upward or 
downward, or leftward or rightward is desired to 
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know, a small splitting number is used: such as 2 x 
2. A storage amount of retrieval information can be 
suppressed. 

[01 03] In a data base where a request frequently aris- 
es for extracting a scene large in motion in a specific 
portion of a picture or two portions spaced apart from 
each other thereof, a comparatively large splitting 
number is used: such as 8 x 8. A highly functional re- 
trieval can be realized. Retrieval information of a high 
resolution includes retrieval information of a low resolu- 
tion; therefore, the retrieval information of a low resolu- 
tion is not necessary to be additionally provided in du- 
plication even when the retrieval information of a low 
resolution is required. 

[01 04] In the above-described embodiment, the num- 
bers of splittings in longitudinal direction and lateral di- 
rection are equal to each other as in cases of 2 x 2, 4 x 
4 and 8x8 and each of the numbers are 2 to the nth 
power, where n is a natural number. When such a split- 
ting number is selectively used, a correspondence can 
be possible between retrieval information from data bas- 
es storing spatially split block unit retrieval information 
having respective different splitting resolutions. Further- 
more, retrieval information having different splitting res- 
olutions can also be compared with each other. 
[0105] Moreover, as shown in Fig. 19 : hierarchical 
storage may be effected of retrieval information in spa- 
tially split block units with respective several levels of 
resolutions. For example, retrieval information of a 
scene includes retrieval information 200 of a scene unit, 
retrieval information 202 in spatially split block units ac- 
cording to 2 x 2 splitting, retrieval information 204 in spa- 
tially split block units according to 4 x 4 splitting and re- 
trieval information 206 in spatially split block units ac- 
cording to 8 x 8 splitting. 

[0106] In the hierarchical structure, retrieval informa- 
tion of a spatially split block unit in a layer is expressed 
with retrieval information in 4 spatially split block units 
of a one-step higher resolution. Therefore, even in a 
case where in each layer, one of retrieval information of 
four spatially split block units is not stored, all the infor- 
mation required for retrieval can be obtained. 
[01 07] Retrieval information according to the example 
shown in Fig. 19 includes retrieval information with the 
same resolution as in storage of retrieval information 
with a spatially split block, as a unit : formed by splitting 
a frame into 8x8 parts, shown in Fig. 17. On the other 
hand, according to retrieval information shown in Fig. 
1 9, an advantage arises that retrieval information of oth- 
er resolutions can be obtained with ease. 
[0108] In the above-described embodiment, all the 
frames of a scene are split in n x n arrangements such 
as 2 x 2, 4 x 4 and 8 x 8 to form spatially split blocks. 
According to the present invention, however, splitting 
methods for a frame are not limited to those. For exam- 
ple, a splitting method may be adopted in which a frame 
is split in ways as shown in Figs. 20 to 24 and retrieval 



information is generated using thus obtained spatially 
split blocks as units. In examples shown in Figs. 20 to 
24, all the frames in a scene are split into 2 to nth power 
(2 n ) (n = 2 in Fig. 20. n = 3 in Fig. 21 , n = 4 in Fig. 22, n 
5 = 5 in Fig. 23 and n = 6 in Fig. 24). Even when such 
splitting methods are used, retrieval information can be 
generated for realizing a similar retrieval function of that 
obtained in an apparatus of the above-described em- 
bodiment. 

10 [01 09] By the splitting methods for a frame as shown 
in Figs. 20 to 24 as well, correspondence between re- 
trieval information having respective different resolu- 
tions is easy in a similar way of the above described em- 
bodiment. Furthermore, easy comparison can be effect- 
's ed between retrieval information having respective dif- 
ferent resolutions. 

[0110] Note that in an example shown in Fig. 5, for 
example, summary information is obtained with frame 
162N as one temporally divided block 164N. However, 

20 the present invention is not limited to such an example. 
For example, one temporally divided block may include 
two ormore frames. Furthermore, it may be adopted that 
each frame is subsampled to form subframes so that 
one temporally split block includes one or a plurality of 

25 subframes. Still furthermore, it may be adopted that 
frames in a scene are partly removed in a proper manner 
and summary information of the remaining frames is 
used. 

[0111] Note that in the examples shown in Figs. 6 and 
30- 8, summary information of a spatially split block is ob- 
tained using all of the split blocks included in the spa- 
tially split block. The present invention, however, is not 
limited to such examples. It is not necessarily required 
to use picture data corresponding to all of the split blocks 
35 in a spatially split block but it is allowed that the split 
blocks are partly removed in a proper manner or picture 
data in each split block is partly removed in a proper 
manner to form summary information of the spatially 
split block. 

40 [0112] Note that when generating retrieval informa- 
tion in temporally divided block units or in spatially split 
block units or retrieval information in scene units, which 
are described above with reference to Figs. 5 and 6, in 
order to remove an influence of motions of a camera 

45 shooting a motion picture on motions of the background, 
retrieval information may be generated, correcting an in- 
fluence due to motion vectors corresponding to the mo- 
tions of a camera. 

[0113] Moreover, in the above-described embodi- 
50 ment, the motion picture retrieval information storage 
apparatus 40 and motion picture retrieving apparatus 50 
are separate apparatuses. Retrieval information is 
transferred from motion picture retrieval information 
storage apparatus 40 to motion picture retrieving appa- 
55 ratus 50 through storage medium 107. Instead of this 
configuration, for example, kinds of retrieval information 
generated by first retrieval information generating sec- 
tion 102, second retrieval information generating sec- 
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tion 103 and third retrieval information generating sec- 
tion 104 5 shown in Fig. 3, may be transferred directly to 
retrieval information managing section 502 of Fig. 1 0 to- 
gether with motion picture structure information output- 
ted from analysis section 101. Retrieval processing in 
this case is a so-called real time retrieval. 
[01 1 4] Split blocks obtained by splitting a frame in the 
above-described embodiment may coincide with encod- 
ed blocks. Blocks of a proper size may be newly defined 
differently from encoded blocks. In the above-described 
embodiment, split blocks are of symmetrical shape and 
arrangement of split blocks is also symmetrical. When 
split blocks are newly defined, however no necessity 
arises for shape of blocks being symmetrical. Further- 
more, no necessity arises for arrangement of split blocks 
being symmetrical either. Still further, partly removal in 
a picture plane can be properly effected. That is, a re- 
gion which is not included in any split block may be 
present in a picture plane. Moreover, split blocks may 
be partly overlapped on each other. That is : a region 
which is included in a plurality of split blocks may be 
present in a picture plane. 

[0115] In the above described embodiment, third re- 
trieval information generating section 104 shown in Fig. 
3 receives summary information outputted from first re- 
trieval information generating section 1 02 and summary 
information outputted from second retrieval information 
generating section 103 to generate summary informa- 
tion of all of a scene. The present invention, however, is 
not limited to such an example. For example, third re- 
trieval information generating section 1 04 may generate 
summary information of all of a scene directly from input 
picture data. 

[0116] In addition, in the above-described embodi- 
ment, various kinds of summary information (retrieval 
information) is obtained with one scene physically de- 
fined as a unit. However, retrieval information is not nec- 
essarily required to be obtained with a scene as a unit. 
For example, retrieval information may be obtained with 
a subscene obtained by dividing one scene into a plu- 
rality of parts, as a unit. That is, the word "scene" also 
means "subscene." To the contrary, retrieval information 
can also be obtained with a composite scene composed 
of a plurality of scenes, as a unit. That is, the word "a 
scene" also means "a collection of plural scenes." 
[01 1 7] Furthermore, while retrieval information is gen- 
erated for one motion picture data and stored and re- 
trieval is executed in the above-described embodiment, 
the present invention is not limited to such an example. 
That is, a case is considered where a plurality of motion 
picture data is adopted as an object for retrieval, retriev- 
al information is generated and stored, and a desired 
scene or a desired frame is retrieved among a plurality 
of pieces of motion picture data. 

[01 1 8] As an example of the case, there is considered 
retrieval of motion picture from a picture data base, 
shown in Fig. 25, in which each of a plurality of scenes 
is stored as motion picture data, independent from the 



other. In a case where shooting of a motion picture is 
performed using a digital camera or the like capable of 
shooting a motion picture, available in recent years, 
each of motion pictures shot, that is motion picture data 
5 expressing one scene of one time picture recording from 
start to end is generated as a data file independent from 
the others. Therefore, the picture data base as shown 
in Fig. 25 can be conceived with ease. 
[0119] Retrieval information is generated by motion 
to picture retrieval information storage apparatus 40 
shown in Fig. 3 for such a picture data base. At this time, 
inputted motion picture data is plural pieces of motion 
picture data present in the data base. Stored picture da- 
ta is already divided into scenes (one motion picture da- 
15 ta includes one scene only); therefore, no necessity aris- 
es for motion picture structure information. Therefore, 
in analysis section 101 , no necessity arises for a struc- 
tural analysis for a motion picture. First, second and third 
retrieval information generating sections 102, 103 and 
20 104 generate respective summary information of a 
frame (a temporally divided block), a spatially split block 
and all of a scene. Fourth retrieval information generat- 
ing section 1 05 generates retrieval information of a mo- 
tion picture, arranging summary information from the 
25 sections into a prescribed format. Storage section 1 06 
receives each motion picture data and retrieval informa- 
tion corresponding to each of the motion pictures to 
store correspondence information between each motion 
picture data and the retrieval information, attaching to 
30- one or both of the motion picture data and the retrieval 
information, or as separate data therefrom, into storage 
medium 107. Storage medium 107 may be provided in 
a data base for the original picture. Moreover, each of 
retrieval information may be stored either independently 
35 from the others or collectively as retrieval information 
corresponding to a picture data base. 
[0120] Furthermore, retrieval is executed by the mo- 
tion picture retrieving apparatus shown in Fig. 10 using 
such a picture data base and retrieval information gen- 
40 erated and stored. At this time, information read out by 
read-out section 501 and managed by retrieval informa- 
tion managing apparatus 502 is retrieval information 
corresponding to a plurality of pieces of motion picture 
data present in a data base and no necessity arises for 
^5 motion picture structure information corresponding to 
each of motion picture data. First, second and third re- 
trieval executing sections 503, 504 and 505 detect a 
scene or a frame meeting a retrieval request using cor- 
responding retrieval information to output information on 
50 the detected scene or frame. Data managing section 
506 and retrieval control section 507 operate in a similar 
manner of the above-described embodiment to obtain 
a retrieval result. 

[01 21] The embodiment disclosed this time should be 
55 construed by way of illustration and example but not by 
way of limitation in all aspects. It is intended that the 
scope of the present invention is not the above-de- 
scribed description but shown by the terms of the claims 
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and includes all of modifications or alterations within the 
claims literally and in the sense of equivalence. 

Industrial Applicability 

5 

[0122] As described above, a motion picture retrieval 
information storage apparatus and a motion picture re- 
trieving apparatus relating to the present invention gen- 
erates retrieval information of a scene with a spatially 
split block obtained not by temporally divided the scene w 
but by spatially splitting the scene, as a unit, to retrieve 
a scene on the basis of the retrieval information; there- 
fore, the apparatuses are suited for use in a system in 
which a desired scene among motion pictures is effi- 
ciently retrieved according to a spatial feature observed is 
through the scene. 

Claims 

20 

1 . A motion picture retrieval information storage appa- 
ratus (40) storing retrieval information into a storage 
medium for retrieving a motion picture, expressed 
by motion picture data, and constituted of one or 
more partial motion pictures (140, 142, 144, 146 ; 25 
160) on the time axis, comprising: 

a retrieval information generating section (1 02 : 
103, 104, 105) generating retrieval information 
corresponding to each of said one or more par- so 
tial motion pictures on the basis of said motion 
picture data; and 

a storage section (1 06) storing said retrieval in- 
formation into a storage medium together with 
correspondence information between said re- 35 
trieval information and said motion picture data : 

wherein said retrieval information generating 
section (1 02, 1 03, 104, 1 05) comprises: a first sum- 
mary information generating means (103) for gen- *o 
erating summary information (188) of one or more 
spatially partial motion pictures obtained by spatial- 
ly splitting each of said one or more partial motion 
pictures (1 40, 1 42, 1 44, 1 46, 1 60) on the time axis. 

45 

2. The motion picture retrieval information storage ap- 
paratus according to claim 1, wherein said one or 
more partial motion pictures (140, 142, 144, 146, 
1 60) on the time axis are obtained by dividing a mo- 
tion picture expressed by said motion picture data 50 
on the time axis, comprising: 

a motion picture structure information output- 
ting section (101) outputting motion picture 
structure information expressing positions, in 55 
said motion picture data ; of said one or more 
partial motion pictures on the time axis, 



wherein said retrieval information generating 
section (102, 103, 104, 105) generates retrieval in- 
formation corresponding to each of said one or 
more partial motion pictures (140, 142, 144, 146. 
160) on the basis of said motion picture structure 
information and said motion picture data, and 

said storage section (106) stores said retriev- 
al information and said motion picture structure in- 
formation into a storage medium together with cor- 
respondence information between said retrieval in- 
formation and said motion picture data. 

3. The motion picture retrieval information storage ap- 
paratus (40) according to claim 1 , wherein 

each of said one or more partial motion pic- 
tures on the time axis comprises: one or more tem- 
poral unit pictures (162A to 162N), and 

said first summary information generating 
means (103) comprises: spatially splitting means 
(1 03) for splitting each of said one or more temporal 
unit pictures (1 62A to 1 62N) of each of said one or 
more partial motion pictures (140, 142, 144, 146. 
1 60) on the time axis into a plurality of spatially split 
blocks in the same manner and generating said 
summary information of a spatially split partial mo- 
tion picture from picture data corresponding to said 
spatially split blocks at the same position on said 
respective one or more temporal unit pictures (1 62A 
to 162N). 

4. The motion picture retrieval information storage ap- 
paratus according to claim 3, wherein 

said spatially splitting means (1 03) splits each 
of said one or more temporal unit pictures ( 1 62A to 
162N) of each of said one or more partial motion 
pictures (140 : 142 : 144, 146, 160) on the time axis 
into the same and predetermined number of parts 
in each of two directions defined on a unit picture to 
form plural spatially split blocks and generates said 
summary information of a spatially split partial mo- 
tion picture from picture data corresponding to said 
spatially split blocks at the same position on said 
respective one or more temporal unit pictures (1 62A 
to 162N). 

5. The motion picture retrieval information storage ap- 
paratus according to claim 3, wherein 

said spatially splitting means (1 03) splits each 
of said one or more temporal unit pictures (1 62A to 
162N) of each of said one or more partial motion 
pictures (1 40, 1 42, 1 44, 1 46, 1 60) on the time axis 
into 2 to the nth power parts, n being a predeter- 
mined number, to form plural spatially split blocks 
and generates said summary information of a spa- 
tially split partial motion picture from picture data 
corresponding to said spatially split blocks at the 
same position on said respective one or more tem- 
poral unit pictures (162A to 162N). 
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6. The motion picture retrieval information storage ap- 
paratus according to claim 5, wherein 

said retrieval information generating section 
(102, 103, 104, 105) further comprises: second 
summary information generating means (102) for 5 
generating summary information of one or more 
temporally partial motion pictures obtained by tem- 
porally dividing said partial motion pictures on the 
time axis by one or more of said one or more tem- 
poral unit pictures (1 62A to 1 62N) included in each 10 
of said one or more partial motion pictures (140 : 
142 : 144, 146, 160) on the time axis, as a unit. 



1 2. The motion picture retrieval information storage ap- 
paratus (40) according to claim 3, wherein 

said retrieval information generating section 
(102, 103 : 104, 105) further comprises: second 
summary information generating means (102) for 
generating summary information of one or more 
temporally partial motion pictures obtained by tem- 
porally dividing said partial motion pictures by one 
or more of said one or more temporal unit pictures 
(1 62A to 1 62N) included in each of said one or more 
partiaJ motion pictures (140, 142, 144, 146, 160) on 
the time axis, as a unit. 



7. The motion picture retrieval information storage ap- 
paratus according to claim 6, wherein '5 

each of said one or more temporal unit pic- 
tures (162A to 162N) comprises a picture of one 
frame. 



1 3. The motion picture retrieval information storage ap- 
paratus (40) according to claim 12, wherein 

each of said one or more temporal unit pic- 
tures (162A to 162N) comprises a picture of one 
frame. 



8. The motion picture retrieval information storage ap- 20 
paratus according to claim 7, wherein 

said retrieval information generating section 
(1 02, 1 03 : 1 04, 1 05) further comprises: a third sum- 
mary information generating section (1 04) generat- 
ing summary information of said partial motion pic- 25 
tures from all of said frames included in each of said 
one or more partial motion pictures ( 1 40 : 1 42, 1 44 : 
146 ; 160) on the time axis. 

9. The motion picture retrieval information storage ap- 30 
paratus according to claim 4, wherein 

said retrieval information generating section 
(102, 103, 104, 105) further comprises: second 
summary information generating means (102) for 
generating summary information of one or more 35 
temporally partial motion pictures obtained by tem- 
porally dividing said partial motion pictures by one 
or more of said one or more temporal unit pictures 
(162Ato 162N) included in each of said one or more 
partial motion pictures (140, 142, 144, 146, 160) on *o 
the time axis, as a unit. 

1 0. The motion picture retrieval information storage ap- 
paratus according to claim 9, wherein 

each of said one or more temporal unit pic- 45 
tures (162A to 162N) comprises a picture of one 
frame. 

1 1 . The motion picture retrieval information storage ap- 
paratus according to claim 10, wherein 50 

said retrieval information generating section 
(102, 103 : 104, 105) further comprises: a third sum- 
mary information generating section (104) generat- 
ing summary information of said partial motion pic- 
tures from all of frames included in each of said one 55 
or more partial motion pictures (140. 142, 144, 146 : 
160) on the time axis. 



14. The motion picture retrieval information storage ap- 
paratus (40) according to claim 13, wherein 

said retrieval information generating section 
(1 02, 1 03, 1 04, 1 05) further comprises: a third sum- 
mary information generating section (1 04) generat- 
ing summary information of said partial motion pic- 
tures from all of said frames included in each of said 
one or more partial motion pictures (140, 142. 144, 
146 : 160) on the time axis. 

1 5. The motion picture retrieval information storage ap- 
paratus (40) according to claim 1 , wherein 

each of said one or more partial motion pic- 
tures (140, 142, 144, 146, 160) on the time axis 
comprises: one or more temporal unit pictures 
(162A to 162N) and 

said retrieval information generating section 
(102, 103 : 104, 105) further comprises: second 
summary information generating means (102) for 
generating summary information of one or more 
temporally partial motion pictures obtained by tem- 
porally dividing said one or more partial motion pic- 
tures (140, 142, 144, 146, 160) on the time axis by 
one or more of said one or more temporal unit pic- 
tures (162A to 162N) included in each of said one 
or more partial motion pictures (140, 142, 144, 146 : 
160) on the time axis, as a unit. 

16. The motion picture retrieval information storage ap- 
paratus (40) according to claim 15, wherein 

each of said one or more temporal unit pic- 
tures (162A to 162N) comprises: a picture of one 
frame. 

1 7. The motion picture retrieval information storage ap- 
paratus (40) according to claim 16, wherein 

said retrieval information generating section 
(1 02, 1 03, 1 04, 1 05) further comprises: a third sum- 
mary information generating section (1 04) for gen- 
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erating summary information of said one or more 
partial motion pictures (140, 142, 144, 146, 160) on 
the time axis from all of frames included in each of 
said one or more partial motion pictures (140. 142, 
144 : 146, 160) on the time axis. 5 

1 8. The motion picture retrieval information storage ap- 
paratus (40) according to claim 15, wherein 

said retrieval information generating section 
(102, 103 ; 104, 105) further comprises: a third sum- w 
mary information generating section (104) generat- 
ing summary information of a partial motion picture 
(140, 142, 144, 146 : 160) on the time axis from all 
of temporal unit pictures (1 62A to 1 62N) included in 
each of said one or more partial motion pictures is 
(140, 142, 144, 146, 160) on the time axis. 

19. The motion picture retrieval information storage ap- 
paratus (40) according to claim 5, wherein 

said retrieval information generating section 20 
(102, 103 : 104, 105) further comprises: a third sum- 
mary information generating section (104) generat- 
ing summary information of a partial motion picture 
(140. 142, 144, 146 : 160) on the time axis from all 
of temporal unit pictures (1 62A to 1 62N) included in 25 
each of said one or more partial motion pictures 
(140. 142, 144. 146, 160) on the time axis. 

20. The motion picture retrieval information storage ap- 
paratus (40) according to claim 4, wherein 30 

said retrieval information generating section 
(102, 103 : 104, 105) further comprises: a third sum- 
mary information generating section (1 04) generat- 
ing summary information of a partial motion picture 
(140, 142, 144, 146, 160) on the time axis from all 35 
of temporal unit pictures (1 62A to 1 62N) included in 
each of said one or more partial motion pictures 
(140, 142, 144, 146, 160) on the time axis. 

21. The motion picture retrieval information storage ap- <*o 
paratus (40) according to claim 1 , wherein 

said retrieval information generating section 
(102, 103 ; 104, 105) further comprises: a third sum- 
mary information generating section (1 04) generat- 
ing summary information of a partial motion picture 45 
(140, 142, 144, 146, 160) on the time axis from all 
of said one or more temporal unit pictures (1 62A to 
162N) included in each of said one or more partial 
motion pictures (140, 142, 144, 146, 160) on the 
time axis. so 

22. The motion picture retrieval information storage ap- 
paratus (40) according to claim 1, wherein 

each of said one or more partial motion pic- 
tures (140, 142, 144, 146, 160) on the time axis 55 
comprises: one or more temporal unit pictures 
(162A to 162N) and 

said retrieval information generating section 
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(1 02, 1 03, 1 04, 1 05) further comprises: a third sum- 
mary information generating section (104) generat- 
ing summary information of a partial motion picture 
(140, 142, 144, 146 : 160) on the time axis from all 
of said one or more temporal unit pictures (1 62A to 
162N) included in each of said one or more partial 
motion pictures (140, 142, 144, 146, 160) on the 
time axis. 

23. A motion picture retrieving apparatus (50) for re- 
trieving a desired picture using retrieval information 
corresponding to each of one or more partial motion 
pictures (140 5 142, 144, 146, 160) on the time axis 
constituting a motion picture, in which motion pic- 
ture data expressing said motion picture is related 
with said retrieval information and said retrieval in- 
formation comprises summary information (1 88) of 
one or more spatially partial motion pictures ob- 
tained by splitting each of one or more partial motion 
pictures (1 40, 1 42, 1 44, 1 46, 1 60) on the time axis, 
comprising: 

information managing means (501, 502) for 
reading and managing said retrieval informa- 
tion; and first partial motion picture retrieving 
means (504), connected to said information 
managing means, for retrieving a partial motion 
picture (140, 142, 144, 146, 160) on the time 
axis meeting first retrieval request in response 
to said first retrieval request with a partial mo- 
tion picture (140, 142, 144, 146, 160) on the 
time axis as a retrieval unit, provided from the 
outside, using said summary information of one 
or more spatially partial motion pictures includ- 
ed in said retrieval information. 

24. The motion picture retrieving apparatus (50) ac- 
cording to claim 23, wherein 

each of said one or more partial motion pic- 
tures (140, 142, 144, 146, 160) on the time axis 
comprises: one or more temporal unit pictures 
(162A to 162N) and 

said retrieval information further comprises: 
summary information of one or more temporally par- 
tial motion pictures obtained by temporally dividing 
said one or more partial motion pictures (140 : 142 : 
1 44. 1 46 : 1 60) on the time axis with one or more of 
said one or more unit pictures (162A to 162N) on 
the time axis included in each of said one or more 
partial motion pictures (140, 142, 144, 146, 160) on 
the time axis, as a unit, 

further comprising: 

second partial motion picture retrieving means 
(505), connected to said information managing 
means (501 , 502), for retrieving a partial motion 
picture (140, 142, 144, 146, 160) on the time 
axis meeting second retrieval request in re- 
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sponse to said second retrieval request with a 
partial motion picture (140, 142, 144, 146 : 160) 
on the time axis as a retrieval unit, provided 
from the outside, using said summary informa- 
tion of one or more temporal partial motion pic- 5 
tures including one or more temporal unit pic- 
tures (162A to 162N) included in said retrieval 
information. 

25. The motion picture retrieving apparatus (50) ac- 
cording to claim 24, wherein 

said retrieval information further comprises: 
summary information with each of said one or more 
partial motion pictures ( 1 40, 1 42, 1 44, 1 46, 1 60) on 
the time axis as a unit, further comprising: 



160) on the time axis, comprising: 

a retrieval information generating section (102 : 
103, 104, 105) generating retrieval information 
corresponding to each of said one or more par- 
tial motion pictures on the basis of said motion 
picture data., and 

said retrieval information generating section 
(102, 103, 104.. 105) comprising: a first summa- 
ry information generating means (1 03) for gen- 
erating summary information (188) of one or 
more spatially partial motion pictures obtained 
by spatially splitting each of said one or more 
partial motion pictures (140, 142, 144, 146, 
1 60) on the time axis. 



third partial motion picture retrieving means 
(503), connected to said information managing 
means (501 , 502), for retrieving a partial motion 
picture (140, 142, 144, 146, 160) on the time 
■ axis meeting third retrieval request in response 
to said third retrieval request with a partial mo- 
tion picture (140, 142, 144, 146, 160) on the 
time axis as a retrieval unit, provided from the 
outside, using said summary information with a 
partial motion picture (140, 142, 144, 146, 160) 
on the time axis as a unit included in said re- 
trieval information. 

26. The motion .picture retrieving apparatus (50) ac- 
cording to claim 23, wherein 

said retrieval information further comprises: 
summary information with each of said one or more 
partial motion pictures (140, 142, 144, 146, 160) on 
the time axis as a unit, 

said apparatus (50) further comprising: 

second partial motion picture retrieving means 
(503), connected to said information managing 
means (501 , 502), for retrieving a partial motion 
picture (140, 142, 144, 146, 160) on the time 
axis meeting second retrieval request in re- 
sponse to said second retrieval request with a 
partial motion picture (140, 142, 144, 146 : 160) 
on the time axis as a retrieval unit, provided 
from the outside, using said summary informa- 
tion with a partial motion picture (1 40, 1 42, 1 44 : 
1 46, 1 60) on the time axis as a unit included in 
said retrieval information. 



Amended claims under Art. 19.1 PCT 



2. (Amended) The motion picture retrieval information 
generating apparatus according to claim 1 , wherein 
said one or more partial motion pictures (140 : 142 : 

20 144 : 146, 160) on the time axis are obtained by di- 

viding a motion picture expressed by said motion 
picture data on the time axis, 

said apparatus further comprising: a motion 
picture structure information outputting section 

25 (101) outputting motion picture structure informa- 

tion expressing positions, in said motion picture da- 
ta, of said one or more partial motion pictures on 
the time axis, 

said retrieval information generating section 

30 (1 02, 1 03, 1 04, 1 05) generates retrieval information 

corresponding to each of said one or more partial 
motion pictures (140, 142, 144, 146, 160) on the 
time axis on the basis of said motion picture struc- 
ture information and said motion picture data. 

35 

3. (Amended) The motion picture retrieval information 
generating apparatus (40) according to claim 1 , 
wherein 

each of said one or more partial motion pic- 
40 tures on the time axis comprises: one or more tem- 

poral unit pictures (162A to 162N) and 

said first summary information generating 
means (103) comprises: spatially splitting means 
(1 03) for splitting each of said one or more temporal 
45 , unit pictures (162A to 1 62N) of each of said one or 
more partial motion pictures (140, 142, 144, 146, 
1 60) on the time axis into plural spatially split blocks 
in the same manner and generating summary infor- 
mation of a spatially split partial motion picture from 
50 picture data corresponding to said spatially split 
blocks at the same position on said respective one 
or more temporal unit pictures (1 62A to 1 62N). 



1 . (Amended) A motion picture retrieval information 

generating apparatus (40) generating retrieval in- 55 
formation for retrieving a motion picture, expressed 
by motion picture data, and constituted of one or 
more partial motion pictures (140, 142, 144, 146 : 



4. (Amended) The motion picture retrieval information 
generating apparatus according to claim 3, wherein 
said spatially splitting means (1 03) splits each 
of said one or more temporal unit pictures (1 62A to 
162N) of each of said one or more partial motion 
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pictures (140, 142, 144, 146, 160) on the time axis 
into the same and predetermined number of parts 
in each of two directions defined on said each unit 
picture to form plural spatially split blocks and gen- 
erates summary information of a spatially split par- 5 
tial motion picture from picture data corresponding 
to said spatially split blocks at the same position on 
said respective one or more temporal unit pictures 
(162A to 162N). 

10 

5. (Amended) The motion picture retrieval information 
generating apparatus according to claim 3, wherein 

said spatially splitting means (1 03) splits each 
of said one or more temporal unit pictures (162A to 
162N) of each of said one or more partial motion '5 
pictures (140, 142, 144, 146, 160) on the time axis 
into 2. to the nth power parts, n being a predeter- 
mined number, to form plural spatially split blocks 
and generates summary information of a spatially 
split partial motion picture from picture data corre- 20 
sponding to said spatially split blocks at the same 
position on said respective one or more temporal 
unit pictures (1 62A to 1 62N). 

6. (Amended) The motion picture retrieval information 25 
generating apparatus according to claim 5, wherein 

said retrieval information generating section 
(102, 103, 104, 105) further comprises: second 
summary information generating means (102) for 
generating summary information of one or more 30 
temporally partial motion pictures obtained by tem- 
porally dividing said one or more partial motion pic- 
tures (140, 142, 144, 146, 160) on the time axis with 
one or more of said one or more temporal unit pic- 
tures (162A to 162N) included in each of said one 35 
or more partial motion pictures (140. 142, 144, 146 : 
160) on the time axis, as a unit. 

7. (Amended) The motion picture retrieval information 
generating apparatus according to claim 6, wherein 40 
each of said one or more temporal unit pictures 
(162A to 162N) comprises: a picture of one frame. 

8. (Amended) The motion picture retrieval information 
generating apparatus according to claim 6, wherein 

said retrieval information generating section 
(1 02, 1 03, 1 04, 1 05) further comprises: a third sum- 
mary information generating means (104) generat- 
ing summary information of a partial motion picture 
(140, 142, 144, 146, 160) on the time axis from all so 
of one or more temporal unit pictures (162A to 
162N) included in each of said one or more partial 
motion pictures (140, 142, 144, 146, 160) on the 
time axis. 



55 



9. (amended) The motion picture retrieval information 
generating apparatus according to claim 4, wherein 
said retrieval information generating section 



(102, 103, 104, 105) further comprises: second 
summary information generating means (102) for 
generating summary information of one or more 
temporally partial motion pictures obtained by tem- 
porally dividing said one or more partial motion pic- 
tures (1 40, 1 42, 1 44, 1 46, 1 60) on the time axis with 
one or more of said one or more temporal unit pic- 
tures (162A to 162N) included in each of said one 
or more partial motion pictures (140, 142, 144, 146 : 
160) on the time axis, as a unit. 

10. (Amended) The motion picture retrieval information 
generating apparatus according to claim 9, wherein 
each of said one or more temporal unit pictures 
(162A to 162N) comprises: a picture of one frame. 

11. Amended) The motion picture retrieval information 
generating apparatus according to claim 9, wherein 
said retrieval information generating section (102, 
103. 104, 105) further comprises: a third summary 
information generating means (104) generating 
summary information of a partial motion picture 
(140, 142, 144, 146, 160) on the time axis from all 
of said one or more temporal unit pictures (162A to 
162N) included in each of said one or more partial 
motion pictures (140, 142, 144, 146, 160) on the 
time axis. 

12. (Amended) The motion picture retrieval information 
generating apparatus (40) according to claim 3, 
wherein said retrieval information generating sec- 
tion (102, 103, 104, 105) further comprises: second 
summary information generating means (102) for 
generating summary information of one or more 
temporally partial motion pictures obtained by tem- 
porally dividing said one or more partial motion pic- 
tures (1 40, 1 42, 1 44, 1 46, 1 60) on the time axis with 
one or more of said one or more temporal unit pic- 
tures (162A to 162N) included in each of said one 
or more partial motion pictures (140, 142, 144, 146. 
160) on the time axis, as a unit. 

1 3. (Amended) The motion picture retrieval information 
generating apparatus (40) according to claim 12 : 
wherein each of said one or more temporal unit pic- 
tures (162A to 162N) comprises a picture of one 
frame. 

14. (Amended) The motion picture retrieval information 
generating apparatus (40) according to claim 13 : 
wherein 

said retrieval information generating section 
(1 02, 1 03, 1 04, 1 05) further comprises: a third sum- 
mary information generating means (104) generat- 
ing summary information of a partial motion picture 
(140, 142, 144, 146, 160) on the time axis from all 
of said one or more temporal unit pictures (162A to 
162N) included in each of said one or more partial 
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35 

motion pictures (140, 142, 144, 146, 160) on the 
time axis. 

15. (Amended) The motion picture retrieval information 
generating apparatus (40) according to claim 1, 5 
wherein 

each of said one or more partial motion pic- 
tures (140, 142, 144, 146. 160) on the time axis 
comprises: one or more temporal unit pictures 
(162A to 162N), and 10 

said retrieval information generating section 
(102, 103, 104, 105) further comprises: second 
summary information generating means (102) for 
generating summary information of one or more 
temporally partial motion pictures obtained by tern- 15 
porally dividing said one or more partial motion pic- 
tures (1 40, 1 42, 1 44, 1 46, 1 60) on the time axis with 
one or more of said one or more temporal unit pic- 
tures (162A to 162N) included in each of said one 
or more partial motion pictures (140, 142, 144, 146 5 20 
160) on the time axis, as a unit. 

16. (Amended) The motion picture retrieval information 
generating apparatus (40) according to claim 15 : 
wherein each of said one or more temporal unit pic- 25 
tures (162A to 162N) comprises: a picture of one 
frame. 

17. (Deleted) 

30 

1 8. (Amended) The motion picture retrieval information 
generating apparatus (40) according to claim 15 : 
wherein 

said retrieval information generating section 
(1 02, 1 03 : 1 04, 1 05) further comprises: a third sum- 35 
mary information generating means (104) generat- 
ing summary information of a partial motion picture 
(140, 142, 144, 146 : 160) on the time axis from all 
of said one or more temporal unit pictures (1 62A to 
1 62N) included in each of said one or more partial <*o 
motion pictures (140, 142, 144, 146, 160) on the 
time axis. 

1 9. (Amended) The motion picture retrieval information 
generating apparatus (40) according to claim 5 : 45 
wherein 

said retrieval information generating section 
(102, 103, 104, 105) further comprises: a third sum- 
mary information generating means (104) generat- 
ing summary information of a partial motion picture so 
(140, 142, 144, 146, 160) on the time axis from all 
of said one or more temporal unit pictures (1 62A to 
162N) included in each of said one or more partial 
motion, pictures (140, 142, 144, 146, 160) on the 
time axis. 55 

20. (Amended) The motion picture retrieval information 
generating apparatus (40) according to claim 4 : 



wherein said retrieval information generating sec- 
tion (102, 103, 104, 105) further comprises: a third 
summary information generating means (1 04) gen- 
erating summary information of a partial motion pic- 
ture (1 40, 1 42, 1 44 s 1 46, 1 60) on the time axis from 
all of said one or more temporal unit pictures (1 62A 
to 1 62N) included in each of said one or more partial 
motion pictures (140, 142, 144, 146, 160) on the 
time axis. 

21 . (Amended) The motion picture retrieval information 
generating apparatus (40) according to claim 3, 
wherein said retrieval information generating sec- 
tion (102, 103, 104, 105) further comprises: a third 
summary information generating means (1 04) gen- 
erating summary information of a partial motion pic- 
ture (140, 142, 144 : 146, 160) on the time axis from 
all of said one or more temporal unit pictures (1 62A 
to 1 62N) included in each of said one or more partial 
motion pictures (140, 142, 144, 146, 160) on the 
time axis. 

22. (Amended) The motion picture retrieval information 
generating apparatus (40) according to claim 1, 
wherein 

each of said one or more partial motion pic- 
tures (140, 142, 144, 146, 160) on the time axis 
comprises: one or more temporal unit pictures 
(162A to 162N), and 

said retrieval information generating section 
(1 02, 1 03, 1 04, 1 05) further comprises: a third sum- 
mary information generating means (104) generat- 
ing summary information of a partial motion picture 
(140, 142, 144, 146, 160) on the time axis from all 
of said one or more temporal unit pictures (1 62A to 
162N) included in each of said one or more partial 
motion pictures (140, 142, 144, 146, 160) on the 
time axis. 

23. A motion picture retrieving apparatus (50) for re- 
trieving a desired picture using retrieval information 
corresponding to each of one or more partial motion 
pictures (140, 142, 144 : 146, 160) on the time axis 
constituting a motion picture, in which motion pic- 
ture data expressing said motion picture is related 
with said retrieval information and said retrieval in- 
formation comprises summary information (1 88) of 
one or more spatially partial motion pictures ob- 
tained by splitting each of one ormore partial motion 
pictures (1 40, 1 42, 1 44, 1 46, 1 60) on the time axis, 

said apparatus (50) comprising: 

information managing means (501, 502) for 
reading and managing said retrieval informa- 
tion; and 

first partial motion picture retrieving means, 
connected to said information managing 
means, for retrieving a partial motion picture 
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(140 : 142, 144, 146, 160) on the time axis meet- 
ing first retrieval request in response to said first 
retrieval request with a partial motion picture 
(1 40 : 1 42, 1 44, 1 46, 1 60) on the time axis as a 
retrieval unit, provided from the outside, using 5 
said summary information of one or more spa- 
tially partial motion pictures included in said re- 
trieval information. 

24. The motion picture retrieving apparatus (50) ac- 10 
cording to claim 23, wherein 

each of said one or more partial motion pic- 
tures (140, 142, 144, 146, 160) on the time axis 
comprises: one or more temporal unit pictures 
(162A to 162N) and 15 

said retrieval information further comprises: 
summary information of one or more temporally par- 
tial motion pictures obtained by temporally dividing 
said one or more partial motion pictures (140, 142, 
1 44 : 1 46, 1 60) on the time axis with one or more of 20 
said one or more unit pictures (162A to 162N) on 
the time axis included in each of said one or more 
partial motion pictures (140, 142, 144, 146, 160) on 
the time axis, as a unit, 

said apparatus (50) further comprising: 25 

second partial motion picture retrieving means 
(505), connected to said information managing 
means (501 , 502), for retrieving a partial motion 
picture (140, 142, 144, 146, 160) on the time 30 
axis meeting second retrieval request in re- 
sponse to said second retrieval request with a 
partial motion picture (1 40, 1 42, 1 44, 1 46, 1 60) 
on the time axis as a retrieval unit, provided 
from the outside, using said summary informa- 35 
tion of one or more temporal partial motion pic- 
tures including one or more temporal unit pic- 
tures (162A to 162N) included in said retrieval 
information. 



25. The motion picture retrieving apparatus (50) ac- 
cording to claim 24, wherein 

said retrieval information further comprises: 
summary information with each of said one or more 
partial motion pictures (140, 142, 144, 146, 160) on 
the time axis as a unit, 

said apparatus (50) further comprising: 

third partial motion picture retrieving means 
(503), connected to said information managing 
means (501 , 502), for retrieving a partial motion 
picture (140, 142, 144, 146, 160) on the time 
axis meeting third retrieval request in response 
to said third retrieval request with a partial mo- 
tion picture (140, 142, 144, 146, 160) on the 
time axis as a retrieval unit, provided from the 
outside, using said summary information with a 
partial motion picture ( 1 40, 1 42, 1 44, 1 46, 1 60) 



on the time axis as a unit included in said re- 
trieval information. 

26. The motion picture retrieving apparatus (50) ac- 
cording to claim 23, wherein 

said retrieval information further comprises: 
summary information with each of said one or more 
partial motion pictures (1 40, 1 42, 1 44, 1 46, 1 60) on 
the time axis as a unit, 

said apparatus (50) further comprising: 

second partial motion picture retrieving means 
(503), connected to said information managing 
means (501 , 502), for retrieving a partial motion 
picture (140, 142, 144, 146, 160) on the time 
axis meeting second retrieval request in re- 
sponse to said second retrieval request with a 
partial motion picture (1 40, 1 42, 1 44, 1 46, 1 60) 
on the time axis as a retrieval unit, provided 
from the outside, using said summary informa- 
tion with a partial motion picture (1 40, 1 42, 1 44, 
1 46, 1 60) on the time axis as a unit included in 
said retrieval information. 

27. (Added) A storage medium (107) in which retrieval 
information for retrieving a motion picture, ex- 
pressed by motion picture data, and constituted of 
one or more partial motion pictures (140, 142, 144, 
146, 160) on the time axis is stored together with 
correspondence information between said retrieval 
information and said motion picture data, 

wherein said retrieval information comprises: 
summary information (188) of one or more spatially 
partial motion pictures obtained by spatially splitting 
each of said one or more partial motion pictures 
(140, 142, 144, 146, 160) on the time axis. 

28. (Added) The storage medium (107) according to 
claim 27, wherein 

each of said one or more partial motion pic- 
tures (140, 142, 144, 146, 160) on the time axis 
comprises: one or more temporal unit pictures 
(162A to 162N) and 

said retrieval information further comprises: 
summary information of said one or more temporal- 
ly partial motion pictures obtained by temporally di- 
viding said one or more partial motion pictures (140, 
142. 144, 146, 160) on the time axis with one or 
more of said one or more temporal unit pictures 
(1 62A to 1 62N) included in each of said one or more 
partial motion pictures (140, 142, 144, 146, 160) on 
the time axis, as a unit. 

29. (Added) The storage medium (107) according to 
claim 27, wherein 

each of said one or more partial motion pic- 
tures (140, 142, 144, 146, 160) on the time axis 
comprises: one or more temporal unit pictures 
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(162A to 162N) and 

said retrieval information further comprises: 
summary information of said one or more temporal- 
ly partial motion pictures generated from all of said 
one or more temporal unit pictures (162A to 1 62N) 
included in each of said one or more partial motion 
pictures (140, 142, 144, 146, 160) on the time axis. 

30. (Added) A motion picture retrieval information man- 
aging apparatus for managing retrieval information 
corresponding to each of one or more partial motion 
pictures (140, 142, 144, 146, 160) on the time axis, 
constituting a motion picture, 

wherein motion picture data expressing said 
motion picture is related with said retrieval informa- 
tion, and 

said retrieval information comprises: summa- 
ry information (188) of one or more spatially partial 
motion pictures obtained by spatially splitting each 
of said one or more partial motion pictures (140 : 
142, 144, 146, 160) on the time axis, 

said apparatus comprising: 

a read-out section (501 ) for reading out said re- 
trieval 

information: and 

a retrieval information managing section (502) 
for holding retrieval information read out by said 
read-out section (501) and for outputting said 
summary information of one or more spatially 
partial motion pictures included in said retrieval 
information in response to a first retrieval re- 
quest with a partial motion picture (140, 142 : 
144, 146, 160) on the time axis as a retrieval 
unit, provided from the outside. 

31 . (Added) The motion picture retrieval information 
managing apparatus according to claim 30, wherein 

each of said one or more partial motion pic- 
tures (140, 142, 144, 146, 160) on the time axis 
comprises: one or more temporal unit pictures 
(162A to 162N), and 

said retrieval information further comprises: 
summary information of one or more temporally par- 
tial motion pictures obtained by temporally dividing 
said one or more partial motion pictures (1 40, 1 42, 
144, 146, 160) on the time axis with one or more of 
said one or more temporal unit pictures (162A to 
162N) included in each of said one or more partial 
motion pictures (140, 142, 144, 146, 160) on the 
time axis, as a unit, and 

said retrieval information managing section 
(502) further outputs summary information of said 
one or more temporally partial motion pictures each 
comprising one or more unit pictures (162A to 
162N) on the time axis included in said retrieval in- 
formation in response to a second retrieval request 
with a partial motion picture (140, 142, 144, 146, 
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160) on the time axis as a retrieval unit, provided 
from the outside. 

32. (Added) The motion picture retrieval information 
managing apparatus according to claim 31 , wherein 

said retrieval information further comprises: 
summary information with each of said one or more 
partial motion pictures (1 40, 1 42, 1 44, 1 46, 1 60) on 
the time axis as a unit, and 

said retrieval information managing section 
(502) further outputs summary information with a 
partial motion picture (140, 142, 144, 146, 160) on 
the time axis as a unit in response to a third retrieval 
request with a partial motion picture (140, 142, 144, 
146, 160) on the time axis as a retrieval unit, pro- 
vided from the outside. 

33. (Added) The motion picture retrieval information 
managing apparatus according to claim 30, wherein 

said retrieval information further comprises: 
summary information with each of said one or more 
partial motion pictures ( 1 40, 1 42, 1 44, 1 46, 1 60) on 
the time axis as a unit, and 

said retrieval information managing section 
(502) further outputs summary information with a 
partial motion picture (140, 142, 144, 146, 160) on 
the time axis as a unit in response to a second re- 
trieval request with a partial motion picture (140, 
142, 144, 146, 160) on the time axis as a retrieval 
unit, provided from the outside. 
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